Performing Scheduled Exchange Server 2003 Monitoring and Maintenance (part 1)

Scheduled Maintenance Tasks

To keep your Exchange organization running smoothly and to provide the level of service that users demand, you need to perform regular, scheduled maintenance tasks. You must define these tasks and the frequency with which you perform them. Typically, you need to perform the following tasks:

• Generate reports and identify trends by using System Monitor, Microsoft Operations Manager, or third-party utilities to capture performance data. This data allows you to identify potential bottlenecks and, for example, to estimate when a server must be upgraded.

• Use protocol logs to track commands that are sent and received by SMTP, NNTP, and HTTP virtual servers. Reviewing these logs can help you troubleshoot messaging problems and identify potential problems.

• Use HTTPMon to monitor your Outlook Web Access (OWA) servers and ensure that client computers are not experiencing connection or performance problems.

• Determine which users consume the most resources on your servers because this information helps you configure and maintain your Exchange storage.

• Monitor the Badmail folder to identify trends and prevent messages from building up. Messages that cannot be delivered and cannot be returned to the sender are sent to the Badmail folder. Over time, these messages can accumulate and use up disk resources.

• Manage the postmaster mailbox, which is used for NDRs. You should define which e-mail account is associated with the postmaster account and monitor and respond to NDR messages when necessary.

Generating Reports and Identifying Trends

Your first task when generating reports and identifying trends is to gather sufficient information to establish a baseline for the performance of each Exchange Server 2003 server. This information allows you to manage your servers proactively and perform trend analysis and capacity planning. You should create a baseline when you first deploy your server, and then re-baseline whenever changes in hardware or usage occur.

To create a baseline and then use deviations from that baseline to track trends and identify potential problems, you must create procedures to monitor your Exchange Server 2003 server. These procedures gather information about server resources such as memory usage, processor utilization, hard disk space utilization, disk performance, and network performance. In addition, performance indicators specific to Exchange Server 2003 server, such as Exchange store performance, message delivery rates, and message queue problems, are included. Your system monitoring procedures should specify the frequency of monitoring tasks, the baseline data to be captured, and the appropriate procedures for managing any problems that may arise.

System measurement is implicit within system monitoring. System measurement parameters include standards for the types of information measured, measurement sampling rates, methods used to analyze data, data storage formats, and reporting formats.

Capacity Planning

Capacity planning enables you to allocate system resources to ensure that optimal system performance is maintained as the system load increases. It is, however, not sufficient to carry out capacity planning using current data, to allocate the appropriate resources, and then to take no further action. Capacity planning is a continuous process that requires that you establish baselines for each service and then monitor all levels of system operations.

The Performance Console

The Performance console is used to gather and display information about performance objects and related counters. The console contains two tools, or snap-ins: Performance Logs And Alerts and System Monitor.

The Performance Logs And Alerts snap-in records and logs system activity over a period of time. The tool enables you to collect data at regular intervals for the counters you select. You can retain logs over extended periods of time by storing data in a database created by Microsoft Access or Microsoft SQL. If you store the data in a database, you can use the reporting features of the database program to create reports that can be used to assess overall performance and to perform trend analysis and capacity planning. Performance Logs And Alerts can also be configured to generate an alert if a counter value either exceeds or drops below a predefined value. The alert, by default, will record an event in Event Viewer, but you can configure it to send an administrative message or to initiate a program.

The System Monitor snap-in lets you chart activity in real time. You also use it to display information captured in log files by the Performance Logs And Alerts snap-in as reports, graphs, or histograms. You can use System Monitor to view server activity whenever server performance degrades. You can, for example, analyze processor activity and queues and use this information to isolate problems with specific components.

Configuring a Performance Console

You should configure the Performance console so that you can determine what is normal system behavior and what modifications you can make to improve performance. You need to know the average number of messages received per user per day and the total number of messages downloaded in a predefined period of time. You also need to know the frequency with which users open folders.

If you have these statistics, you can calculate the number of additional users that each of your servers can support. However, this is anything but a straightforward calculation. E-mail traffic is not smooth, but rather is notoriously peaky. You also need to know the peak delivery rate, the peak period during the day, the peak day of the week, and whether monthly or quarterly peaks exist.

To obtain this information, you need to create a Performance console that allows you to see the entire system environment and that registers changes in the performance of your servers. The guidelines for creating a Performance console are as follows:

• Create a Performance console that has two different sample times, for example:

  • 900 seconds for a 24-hour view

  • 10 seconds to catch short-lived spikes

• Include a minimal set of counters in each console, for example:

  • Memory\Pages/sec

  • Processor(_Total)\% Processor Time

  • Process(store)\% Processor Time

  • MSExchangeIS\RPC Requests

  • MSExchangeIS\RPC Operations/sec

  • PhysicalDisk(_Total)\Disk Transfers/sec

  • SMTP Server\Local Queue Length

  • SMTP Server\Messages Delivered/sec

  • MSExchangelS Mailbox\Local Delivery Rate

  • MSExchangelS Mailbox\Folder Opens/sec

  • MSExchangelS Mailbox\Message Opens/sec

• Examine your busiest server to understand why it is busy and what performance problems can be resolved.

• Save your reference log files so that you can develop historical baseline data that allows you to see what changes have occurred and to accommodate for additional growth over time.

Analyzing Trends

By analyzing the reports you create using your data, you can uncover patterns and predict future trends. This trend analysis assists you to determine what steps you can take to prevent problems on your Exchange Server 2003 servers in the future. For example, you can predict when normal growth, such as mailbox growth, will require you to upgrade your storage.

Protocol Logs

Protocol logs were introduced, and protocol logging configured. You can use protocol logging to track commands that an HTTP, SMTP, or NNTP virtual server receives from client computers and to track outgoing commands. You should establish a schedule for reviewing protocol logs, and you should also review the log files if your users are experiencing problems.

You can select from four types of file formats, depending on how you want to store the information. Table 1 describes these formats.

You enable protocol logging and select the log file format for an SMTP or NNTP virtual server by using Exchange System Manager to configure the virtual server properties. You can also specify the schedule for creating new log files and the location of these files (except for ODBC format). If your log file format is W3C Extended Log File Format, you can select the items that you want to track.

You manage HTTP protocol logging by using the IIS Manager console to configure the Web site properties. By default, protocol logging is enabled, and the log file format is set to W3C Extended Log File Format.

Using a Protocol Log File

You can use a protocol log file for general troubleshooting. Suppose, for example, users report that they are receiving messages when their addresses do not appear on the To or Cc lines. In this case, you can use SMTP log file data to compare the recipients specified in a rcpt to command with addresses posted in a message header or in the To and Cc lines of the message. Another example is when you search the SMTP log file for the message identity (ID) of a remote system so that you can collaborate with another system administrator to trace a message.

You can look for response codes that a receiving server returns after your server issues an ehlo command to determine the maximum-size message that a server will accept (for example: 250-size 60000000). You can also use the log data to generate reports. If another server attempts to use your server as a relay (assuming your server is properly configured to prevent unauthorized SMTP relaying), the log file posts a numeric response code of 550, which corresponds to Relaying Prohibited, in the protocol status (sc-status) field. You could write a script to search this field and tally the number of reported 550 codes.

HTTP Monitor

You can use the HTTP Monitor (HTTPMon) Resource Kit utility to monitor Web sites or applications. HTTPMon can check Web sites and report the results to either a log file in comma-separated values (CSV) format or in the Windows Server 2003 server event log. You can then either use Windows Management Instrumentation (WMI) to monitor the event log or import the CSV output into a Microsoft Excel or SQL database for further analysis.

The utility enables you to test that the Exchange Web site is responding to requests from client computers in a timely fashion. It lets you test several sites simultaneously to ensure that they are up and that they are responding within reasonable times. You can use HTTPMon to monitor OWA servers and identify and troubleshoot problems. You should establish a schedule for monitoring OWA on a regular basis.

HTTPMon comprises the following three components:

• Realtime Sampling Service The real-time monitoring service.

• SQL Reporting Server Pulls data from monitor servers and loads it into Microsoft SQL Server.

• Client Monitor A set of Web pages that displays the results from the SQL Reporting Server database.

Installing HTTPMon

Currently, the HTTP Monitoring Tool is available only in the Windows 2000 Resource Kit and is not installed by the Resource Kit Setup program. To install it, insert the Windows 2000 Resource Kit companion CD into your CD-ROM drive. When the Setup screen appears, click Explore The CD. In the <cdroot>\apps\httpmon directory, double-click Setup.exe and follow the directions that appear on your screen.

When you install the HTTP Monitoring Tool, the information file (Readme.txt) and the user’s guide (HTTPMon_whitepaper.doc) are also installed. Access these files for information on configuring and using the tool. They are both located in the directory where you install the tool (C:\Program Files\Httpmon in the default installation).

Mailbox Limits

You need to manage mailbox limits on a regular basis. By default, Exchange Server 2003 permits large mailbox sizes, and this allows users to store a great deal of information in their mailboxes. However, this can lead to some users having excessively large mailboxes. You can use Exchange System Manager to manage mailbox limits and Event Viewer to check whether mailbox sizes have reached the various limit stages that you configured.

If you access the mailbox node in Exchange System Manager, then you can modify the mailbox view to include the Storage Limits column. This column provides feedback on whether certain limits have been enforced for a specific mailbox and, if so, what condition the mailbox is in. You can then use Event Viewer to view the events recorded when a mailbox reaches a limit threshold. Microsoft recommends that you monitor mailbox sizes at least once per week. Table 2 describes the limit settings that you can specify in the Storage Limits column.

You can configure diagnostic logging on an Exchange Server 2003 server to enable you to view events in the application log in Event Viewer that indicate when mailboxes reach the various stages of storage limit warnings. You can also use the mailbox management process, otherwise known as Mailbox Manager, to manage oversized mailboxes.

Mailbox Manager is defined as a recipient policy and can be used to create reports or take actions to clean old mail from users’ mailboxes. You can schedule the mailbox management process to run at a specific time for regularly scheduled cleanups, or you can run it manually. You should run the process manually if you are in immediate need of mailbox statistics or if you need to clear mailboxes of old mail immediately.

You start the process by starting Exchange System Manager, navigating to the relevant server, right-clicking the server, and then selecting Start Mailbox Management Process. Mailbox management starts after a short delay, which depends on the level of resource use on the server.

The Badmail Folder

If a message has reached the retry limit and an NDR cannot be delivered to the sender, a copy of the message is placed in the Badmail folder. Messages placed in this folder cannot be delivered or returned.

Over time, messages can accumulate in the Badmail folder. If you do not monitor the Badmail folder regularly, your Exchange Server 2003 server can run out of disk space. This causes the Microsoft Exchange Information Store service to shut down. You need to establish policies for monitoring the folder and removing messages after a certain age. Microsoft recommends reviewing the content of the Badmail folder on a weekly basis.

Another reason for monitoring the Badmail folder is that it could contain messages from an external user who is attempting to relay spam e-mail through your Exchange organization.

By default, the Badmail folder is located in the virtual server’s home directory. You can relocate the folder by using Exchange System Manager to access the Messages tab in the relevant virtual server’s Properties dialog box. You can use Windows Explorer to check the contents of the Badmail folder. A large number of undelivered messages could indicate delivery problems such as a DNS or network failure, or potential security problems due to spam e-mail. You should delete messages from the Badmail folder based on your organization’s policies. For example, you could delete messages once per week.

The Postmaster Mailbox

The postmaster account is configured to receive NDRs and is used to send the delivery status of a message to the sender. By default, Exchange Server 2003 creates the postmaster proxy address and assigns this address to the administrator who created the Exchange organization. It is good practice to change this default setting so that the name of this administrator account is not exposed to outside users.

To designate a specific user’s mailbox as the postmaster mailbox for a local SMTP domain, you add the proxy postmaster@localdomainname to the user’s list of SMTP proxy addresses. You can associate an existing e-mail account with the postmaster or create a dedicated postmaster account from which the NDRs will be sent.

If you decide to create a dedicated postmaster account, you can log on using that account by using an Outlook profile and respond to the account messages. Alternatively, you can delegate Send As permissions on the account to the person to whom you have delegated the task of managing the mailbox, and add the mailbox to that person’s Outlook profile.

You should establish a regular schedule for reviewing and responding to the delivery reports contained in the postmaster mailbox. This schedule should be based on your organization’s requirements. Some organizations make this a daily task in an effort to reduce the number of e-mail messages that are delivered to users who are no longer with the organization, while others companies make this a weekly or monthly maintenance task.